Papers with magnitude-based pruning
DeFT-X: Denoised Sparse Fine-Tuning for Zero-Shot Cross-Lingual Transfer (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Prior studies have explored multiple approaches to combine task knowledge from task-specific data in a (high-resource) source language with language knowledge from unlabeled text in 'low-resourced' target language. |
| Approach: | They propose a composable sparse fine-tuning approach that learns task-specific and language-specific sparsen masks to select a subset of the pretrained model's parameters. |
| Outcome: | The proposed approach performs at par or outperforms SFT and other prominent cross-lingual transfer baselines. |
Dual Activation-Weight Sparsity: A Training-Free Framework for Efficient Large Language Model Compression (2026.acl-long)
Copied to clipboard
Luoyang Sun, Guangyan Li, Cheng Deng, Haifeng Zhang, Jian Zhao, Yongqiang Tang, Wensheng Zhang, Jun Wang
| Challenge: | Large language models (LLMs) excel at natural language tasks but face deployment bottlenecks due to computational demands. |
| Approach: | They propose a training-free framework that exploits activation and weight sparsity . they use a three-tier routing strategy that uses magnitude-based pruning . |
| Outcome: | Experiments on Llama and Mistral models show that DAWS outperforms activation-weight sparsity pruning methods. |
Learning to Win Lottery Tickets in BERT Transfer via Task-agnostic Mask Training (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent studies show pre-trained language models contain matching subnetworks that have similar transfer learning performance as the original PLM. |
| Approach: | They propose to prune matching subnetworks using magnitude-based pruning . they propose to optimize the subnetwork structure towards the pre-training objectives . |
| Outcome: | The proposed method is more efficient in searching subnetworks and advantageous when fine-tuning within a range of data scarcity. |